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INFORMATION  DETAIL  AND  DISPLAY  CONCEPTS  FOR  CRITICAL  DECISIONS  IN 
BALLISTIC  MISSILE  DEFENSE  COMMAND  AND  CONTROL 


INTRODUCTION 

Commanders  and  operators  in  command  and  control  (C2)  centers  for  theater-level  ballistic 
missile  defense  (BMD)  are  responsible  for  monitoring  a  largely  automated  engagement  process. 
Their  actions  include  monitoring  the  current  situation,  as  well  as  intervening  in  near-real  time 
during  anomalous  situations  or  when  new  information  is  received.  The  displays  for  these  missile 
defense  systems  need  to  support  the  human  operators  in  performing  these  actions.  Displays  can 
accomplish  this  by  ensuring  that  critical  information  items  needed  to  monitor  a  situation  are 
always  displayed  and  that  detailed  or  amplified  information  to  support  this  monitoring  or  to 
intervene  when  needed  is  available  to  the  operators. 

The  theater  high  altitude  area  defense  (THAAD)  system  is  a  missile  defense  system  being 
developed  for  the  United  States  Army.  Previous  U.S.  Army  Research  Laboratory  (ARL)  studies 
and  experiments  have  been  conducted  to  derive  the  C2  information  requirements  for  this  system 
(Knapp,  Ensing,  &  Reynolds,  1995).  These  studies  addressed  the  issues  of  (a)  information 
categorization,  (b)  attention  direction  and  focusing,  and  (c)  information  criticality.  Results  from 
the  experiments  have  given  designers  important  information  regarding  interface  display  design  for 
the  THAAD  system.  In  particular,  these  results  have  told  designers  (a)  what  information  areas 
were  critical  and  needed  to  be  presented  at  a  high  level  in  the  display,  and  (b)  what  information 
items  within  these  information  areas  were  critical  and  needed  to  be  displayed  in  a  prominent 
manner. 

In  the  THAAD  system,  an  information  area  is  often  displayed  as  its  own  individual 
window  (e.g.,  threat  engagement  queue  and  track  amplification  windows).  In  designing  these 
windows,  designers  need  to  know  what  information  is  critical  and  needs  to  be  displayed  on  the 
first  page  of  the  window.  Information  that  is  less  critical  or  unimportant  can  then  be  delegated  to 
subsequent  window  pages,  accessible  to  the  operator  via  scrolling,  or  to  a  lower  display  level, 
accessible  to  the  operator  via  mouse  clicks. 

OBJECTIVE 

The  objective  of  the  experiment  described  here  was  to  determine  which  information  items 
in  the  engagement  queue  and  track  amplification  windows  of  the  THAAD  operator  system 
interface  (OSI)  are  critical  enough  to  demand  being  displayed  in  prominent  locations  within  the 
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windows.  Because  different  information  items  are  critical  in  different  situations,  two  situations 
were  presented  for  each  window  to  determine  which  items  are  critical  in  several  representative 
tasks.  Before  this  experiment  is  described,  however,  a  brief  explanation  of  the  task  domain, 
ballistic  missile  defense,  and  the  THAAD  OSI  follows. 

BACKGROUND:  BALLISTIC  MISSILE  DEFENSE  AND  THE  THAAD  OSI 

Ballistic  missile  defense  incorporates  the  means  needed  to  intercept  enemy  ballistic 
missiles  deployed  against  friendly  forces  or  assets.  Theater  missile  defense  takes  this  a  step 
further  by  requiring  the  defense  to  be  “portable”  and  flexible  enough  to  be  deployed  rapidly  to 
different  theaters  or  different  locations  within  a  theater.  THAAD  is  being  developed  to  work 
with  the  Patriot  missile  system  to  provide  such  a  defense.  They  are  intended  to  provide  a  two- 
tiered  defense  in  order  to  provide  a  more  “leakproof’  defense — ^THAAD  at  the  upper  level  and 
Patriot  at  the  lower  level.  Together,  these  two  systems  will  be  able  to  defend  a  large  area, 
increase  defense  effectiveness  by  destroying  targets  farther  away  from  assets,  and  provide  for 
multiple  shot  opportunities  (Kilgore,  1994). 

The  THAAD  OSI  prototype  has  been  used  to  develop  the  design  features  for  the  actual 
THAAD  tactical  operations  center  (TOC).  Version  2.1  of  the  prototype  software  is  being 
revised.  This  version  allows  users  to  perform  some  initial  force  operations  (FO)  tasks,  needed 
for  current  operations  management,  and  engagement  operations  (EO)  tasks,  needed  for  current 
battle  (engagements)  management.  The  main  focus  of  this  current  work  is  on  engagement 
operations,  so  this  description  is  limited  to  those  functions  (see  Litton  Systems,  Inc.,  1994,  for 
more  information).  During  EO,  the  operator  can  manage  engagements  of  tactical  ballistic  missiles 
(TBMs),  monitor  the  air  situation,  and  control  launcher  and  sensor  operations.  Two  engagement 
modes  can  be  chosen:  automatic,  when  the  system  automatically  launches  THAAD  interceptors 
in  response  to  threats;  and  semiautomatic,  when  the  operator  needs  to  authorize  all  interceptor 
laimches. 

One  of  the  main  features  of  the  current  OSI  version  is  an  air  display  map,  which  displays 
a  graphical  view  of  the  area  under  THAAD’s  protection,  as  well  as  the  incoming  TBMs  (threats) 
and  any  outgoing  THAAD  interceptors.  Other  features  include  a  menu  bar  at  the  top  of  the 
screen  and  a  status  bar  at  the  bottom  left  of  the  screen.  The  rest  of  this  section  describes  two 
windows  accessible  from  the  menu  bar:  the  engagement  queue  window  and  the  track 
amplification  window. 
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ENGAGEMENT  QUEUE 


Figure  1.  Tabidar  engagement  queue  window. 


Tabular  track  amplification  window. 


Apparatus  and  Stimuli 

The  study  was  conducted  using  a  series  of  paper-and-pencil  comparative  judgment  tasks. 
Each  task  consisted  of  a  written  situation  or  scenario  for  the  subjects  to  read  and  a  representation 
of  a  window  for  the  engagement  queue  or  track  amplification  window  in  the  THAAD  OSI. 

Inside  the  windows  were  listed  the  information  items  currently  in  the  engagement  queue  window 
or  track  amplification  window  (drawn  from  the  OSI  Prototype  Version  2.1).  Ten  items  were 
listed  inside  an  engagement  queue  “window,”  while  18  to  25  items  were  listed  in  a  track 
amplification  “window”;  the  number  of  amplification  items  varied  according  to  whether  the 
operator  was  vieiving  an  unengaged  track  (18  items  shown  in  the  window)  or  an  engaged  track  (25 
items  shown  in  the  window).  Also  included  was  a  practice  drill  in  which  the  operators  viewed 
the  “map  utilities”  window.  The  practice  drill  was  included  to  allow  the  subjects  to  become 
comfortable  with  the  experimental  task. 


Procedure 

The  method  used  for  refining  the  items  displayed  in  the  engagement  queue  and  track 
amplification  windows  for  the  THAAD  OSI  was  based  on  that  determined  by  Knapp  et  al. 
(1995).  The  original  method  for  determining  information  criticality  involved  using  the  method  of 
paired  comparisons.  Once  it  was  determined  which  information  items  were  grouped  together 
logically  (forming  information  clusters)  and  which  information  clusters  were  critical  to  particular 
decision  tasks,  then  each  of  the  informatipn  items  in  an  information  cluster  was  paired  together 
with  the  other  items  and  rated  as  to  which  item  in  the  pair  was  more  important  for  each  decision 
task. 

The  paired  compmsons  method  has  given  valid  and  reliable  results.  However,  for  this 
experiment,  the  number  of  information  items  in  the  engagement  queue  (10)  and  track 
amplification  (18  through  25)  clusters  would  require  too  many  comparisons  (45  and  153  through 
300,  respectively)  to  be  made  comfortably  within  the  confines  of  one  experiment.  Therefore,  this 
method  of  comparison  was  modified  so  that  each  subject  was  presented  with  a  static  engagfttnftnt 
queue  (or  track  amplification)  “window”  for  two  decision  task  situations:  (a)  general  situation 
monitoring  and  assessment,  and  (b)  assessing  impact  of  new  intelligence — threat  capabilities  H^ta, 
Inside  each  window  was  listed  each  of  the  information  items  in  the  cluster  (10  for  the  engagement 
queue  and  1 8  through  25  for  the  track  amplification).  The  subjects  were  instructed  to  read  the 
decision  task  and  quickly  compare  all  items  to  determine  the  items  that  were  the  most  important 
to  them  in  addressing  the  situation,  and  which  items  were  unnecessary  or  unimportant.  The 
subjects  were  then  instructed  to  select  the  three  most  important  and  three  least  important  items 
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for  the  engagement  queue,  and  the  five  most  important  and  five  least  important  items  for  the  track 
amplification  window. 

Each  subject  received  the  experimental  situation  sheets  in  the  same  order.  However,  the 
items  presented  in  the  engagement  queue  and  track  amplification  windows  were  in  a  randomized 
order. 

Each  subject  first  received  a  practice  drill  and  was  told  to  read  the  given  situation.  He  or 
she  was  then  given  20  seconds  to  scan  the  items  in  the  map  utilities  window,  determine  which 
three  items  they  were  likely  to  use  first,  and  to  circle  these  three  items  with  a  pen  or  pencil 
(provided). 

After  the  practice  drill,  each  subject  received  a  sheet  of  paper  with  two  situations  asking 
him  or  her  to  use  the  THAAD  engagement  queue  window.  The  subjects  were  asked  to  read 
Situation  1  (involving  situation  assessment  and  monitoring)  and  were  then  given  20  seconds  to 
scan  the  engagement  queue  window,  determine  the  top  three  items  they  would  consult  first,  to 
circle  these  items,  determine  the  very  last  three  items  they  would  use  or  could  do  without,  and  to 
cross  out  those  three  items.  Subjects  were  then  asked  to  read  Situation  2,  involving  imexpected 
threat  capabilities.  They  were  then  given  the  same  directions  regarding  the  engagement  queue 
window  and  the  items  in  that  window. 

After  completing  the  tasks  involving  the  engagement  queue,  subjects  were  given  three 
sheets  of  paper  with  tasks  involving  the  track  amplification.  Subjects  were  told  to  read  Situation 
1  (on  the  first  sheet),  involving  situation  awareness  of  an  unengaged  track.  They  were  then  told 
they  had  60  seconds  to  scan  the  items  in  the  track  amplification  window,  to  determine  which  five 
items  would  be  most  iiseful  in  that  situation,  to  circle  those  items,  to  determine  which  five  items 
they  did  not  really  need,  and  to  cross  out  those  five  items. 

The  second  sheet  of  paper  involved  the  same  situation  (Situation  1),  but  with  the 
operator  viewing  an  engaged  track.  Participants  received  the  same  directions  to  select  the  five 
most  xisefiil  and  the  five  “not  needed”  items. 

The  last  sheet  of  paper  each  subject  viewed  contained  Situation  2  for  the  track 
amplification,  having  to  do  with  unexpected  new  threats  and  the  impact  on  an  engaged  track. 
Participants  then  received  the  same  directions  to  select  the  five  most  useful  and  five  least  useful 
items. 
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Data  Analysis 

The  resulting  data  were  then  tabulated  into  a  matrix  for  each  participant.  An  item 
indicated  as  being  important  was  given  a  “score”  of +1,  an  item  indicated  as  being  unimportant 
was  given  a  “score”  of  -1,  and  the  remaining  items  received  a  score  of  0.  These  scores  were  then 
summed  across  all  participants  in  each  of  the  two  groups  and  recorded  in  a  matrix  of  data  item 
scores  for  each  situation  (first  for  the  engagement  queue  and  then  for  the  track  amplification 
window).  The  resulting  matrices  indicate  which  items  could  be  considered  important  and  which 
tmimportant  (with  importance  being  determined  by  positive  total  scores)  for  each  of  the 
windows.  To  make  the  items’  rating  summary  score  directly  comparable,  differences  in  sample 
sizes  between  the  military  user  group  and  the  Litton  design  engineering  group  were  normalized 
by  conversion  to  item  group  rating  percentages.  For  example,  if  four  of  the  five  soldiers  rated 
“method  of  fire”  as  important,  then  80%  of  the  group  agreed.  A  group  agreement  value  of  60% 
was  used  as  the  criterion  to  determine  whether  an  item  was  critically  important. 

RESULTS 
Engagement  Queue 

Situation  1  (situation  assessment  and  monitoring) 

Looking  at  Figure  3,  four  information  items  were  indicated  as  important  in  the 
engagement  queue  wdndow  for  Situation  1  by  the  military  participants  (asset  thr^atpinpH  TBM 
type/subtype,  time  to  ground  impact/point,  and  time  to  earliest  launch  [TEL]).  Referring  to 
Figure  4,  of  these  four,  one  item  was  deemed  critical  by  at  least  60%  of  the  military  group  for  a 
general  situation  assessment  and  monitoring  sitnatinn. 

Again  referring  to  Figure  3,  the  design  engineers  rated  three  information  items  as 
important  for  the  engagement  queue  during  Situation  1.  These  items  were  asset  threatened, 
engagement  status  and  time  to  last  launch  (TLL).  Based  on  the  presentation  in  Figure  4,  all  three 
items  were  rated  as  critical  by  at  least  60%  of  the  engineering  group. 

The  lower  half  of  Figure  3  diows  which  information  items  were  rated  as 
unimportant  for  general  monitoring.  For  the  military  user  group,  two  items  in  the  pngag^ment 
queue  (method  of  fire  and  lower  tier  coverage)  were  found  to  have  consistently  low  priority. 

Both  of  these  information  items  were  rated  as  having  low  priority  by  at  least  60%  of  the  military 
group  (see  Figure  4,  lower  halQ. 

Design  engineers  rated  four  items  (method  of  fire,  lower  tier  coverage,  time  to 
intercept,  and  TEL)  as  the  least  important  items  for  a  general  assessment  and  monitoring  situation. 
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Percent  of  Sample  Rating  Item 
Critical  {High  vs  Low) 


Engagement  Queue  Item  Criticality:  Situation  One 
General  Situation  Assessment  and  Monitoring 


#  Military  Users 
m  Design  Engineers 

A.  Both 

Fiyure  3.  Data  matrix  for  the  engagement  queue  (Situation  1). 
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Percent  of  Sample  Rating  Ite 
Critical  (High  vs  Low) 


Engaement  Queue  Information  Item  Ratings:  Situation  One 
General  Situation  Assessment  and  Monitoring 


9  Military  Users 
SI  Design  Engineers 

Both 

Figure  4.  Data  matrix  for  the  engagement  queue  group  percentages  (Situation  1). 
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Figure  4  shows  that  two  items  (method  of  fire  and  lower  tier  coverage)  received  low  ratings  by  60% 
of  the  group. 

Situation  2  (new  intelligence  about  unexpected  threat  capabilities) 

Looking  at  Figure  5^  four  information  items  were  indicated  as  important  in  the 
engagement  queue  window  for  Situation  2  (asset  threatened,  TBM  type/subtype,  time  to  intercept, 
and  threat  priority).  Figure  6  reveals  that  three  of  these  fom  were  deemed  critical  by  at  least  60%  of 
the  military  participants.  These  critical  items  were  asset  threatened,  TBM  type/subtype,  and  threat 
priority. 

Figure  5  shows  that  design  engineers  rated  two  information  items  as  important  to 
“responding  to  new  intelligence  about  threat  capabilities”  (Situation  2).  These  items  were  asset 
threatened  and  TLL,  both  of  which  exceed  the  60%  criterion  (see  Figure  6). 

The  lower  half  of  Figure  5  shows  which  information  items  were  rated  as  unimportant 
for  Situation  2.  For  the  military  user  group,  six  items  in  the  engagement  queue  (method  of  fire,  time 
to  grotmd  impact/point,  status  [of  the  engagement],  TEL,  lower  tier  coverage,  and  TLL)  were  found 
to  have  consistently  low  priority  in  this  “new  intelligence  information”  situation  by  military  users. 
As  Figure  6  shows,  two  of  these  items  (status  and  lower  tier  coverage)  were  rated  as  having  low 
priority  by  at  least  60%  of  the  military  user  group. 

For  this  same  situation,  design  engineers  rated  four  items  (method  of  fire,  threat 
priority,  time  to  ground  impact/point,  and  lower  tier  coverage)  as  the  least  important.  Figure  6 
indicates  that  method  of  fire,,  time  to  ground  impact/point,  and  lower  tier  coverage  were  rated  as 
having  low  priority  by  at  least  60%  of  the  group. 

Stmimaiy 

Figure  7  presents  a  summary  of  the  military  group  agreement  percentages 
displayed  in  Figures  4  and  6.  Figrrre  8  summarizes  the  design  group  percentages  presented  in  Figures 
4  and  6.  For  Figures  7  and  8,  plus  and  minus  signs  in  firont  of  percentage  values  are  only  to  indicate 
relative  importance.  They  carry  no  mathematical  significance.  A  plus  sign  denotes  relative 
importance  whereas  a  minus  sign  symbolizes  relative  unimportance.  Rank  order  values  were  derived 
fi-om  the  average  group  percentage  ratings  of  all  situations  rated.  The  rank  order  information  is 
addressed  later  in  the  Discussion  section.  No  information  item  rated  as  critical  by  soldiers  for  the 
engagement  queue  window  were  common  to  both  of  the  situations  tested.  For  BMC3  system  design 
engineers,  asset  threatened  and  TLL  were  rated  as  important  by  at  least  60%  of  the  group  across 
both  situations. 
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Percent  of  Sample  Rating  Item 
Critical  (High  vs  Low) 


Engagement  Queue  Item  Criticality:  Situation  Two 
Respond  to  New  Intelligence  About  Threat  Capabilities 


#  Military  Users 
|T|]  Design  Engineers 

^  Both 

Figure  5.  Data  matrix  for  the  engagement  queue  (Situation  2). 
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Percent  of  Sample  Rating  Item 
Critical  (High  vs  Low) 


Engagement  Queue  Information  Item  Ratings;  Situation  Two 
Respond  to  New  Intelligence  About  Threat  Capabilities 


#  Military  Users 


r~~l  Design  Engineers 
My  Both 


Figure  6.  Data  matrix  for  the  eng^ement  queue  grovq)  percentages  (Situation  2). 
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Situation  1 

Situation  2 

Rank  Order 

Method  of  Fire 

-0.6 

-0.2 

9 

Asset  Threatened 

+.4 

+.8 

1 

Time  to  Intercept 

0 

+.2 

5 

Threat  Priority 

0 

+.6 

3 

TBM  Tvpe/Subclass 

+.4 

+.6 

2 

Time  to  Ground  IMP 

+.2 

-0.2 

6 

Status 

0 

-0.6 

8 

Time  to  Earliest 

Launch 

+.6 

-0.2 

4 

Lower  tier  Coverage 

-1 

1 

o 

10 

TLL 

0 

-0.4 

7 

Figure  7.  Engagement  queue  -  information  item  importance  rating  group  agreement  percentages 
for  military  users. 


Situation  1 

Situation  2 

Method  of  Fire 

-0.66 

-0.66 

9 

Asset  Threatened 

+1.0 

1 

Time  to  Intercept 

-0.33 

0 

6 

0 

-0.33 

6 

TBM  Tvpe/Subclass 

0 

+.33 

4 

Time  to  Ground  IMP 

0 

8 

Status 

+.66 

0 

3 

Time  to  Earliest 
Launch 

-0.33 

0 

6 

Lower  tier  Coverage 

-1 

-0.66 

10 

TLL 

+.66 

+1.0 

2 

Figure  8.  Engagement  queue  -  information  item  importance  rating  group  agreement  percentages 
for  design  engineers. 


Of  the  information  items  in  the  engagement  queue,  two  were  indicated  by  the 
military  group  as  having  low  priority  across  both  situations  tested  (method  of  fire  and  lower  tier 
coverage).  Of  these  two,  only  one  (lower  tier  coverage)  was  found  to  have  low  priority  across 
both  situations  by  at  least  60%  of  the  military  group. 

Sixty  percent  of  the  design  group  rated  rnethod  of  fire  and  lower  tier  coverage  as 
least  important  across  both  Situations  1  and  2. 
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Across  both  groups  and  both  situations,  there  was  agreement  by  at  least  60%  of 
each  group  on  one  item  as  least  important  (lower  tier  coverage),  but  60%  of  both  groups  did  not 
concur  on  any  item  as  most  important. 


Track  Amplification 

Situation  lA  (situation  assessment  and  monitoring  for  imengaged  tracks) 

Looking  at  Figure  9,  eight  items  were  indicated  as  important  in  the  track 
amplification  window  for  Situation  1 A  for  military  users.  These  items  included  asset  threatened, 
threat  priority,  engagement  status,  TLL,  speed  of  track,  position  of  track,  heading  of  track,  and 
classification/ID.  Of  these  eight,  Figure  10  indicates  that  two  items  (engagement  status  and 
classification/ID)  were  deemed  critical  by  at  least  60%  of  the  military  participants. 

The  lower  half  of  Figure  9  indicates  that  eight  items  in  the  track  amplification 
window  (method  of  fire,  lower  tier  coverage,  estimated  launch  point,  radar/source  unit,  trajectory 
profile/pop-up  graphic,  track  quality,  asset  damage  score,  and  altitude  of  track)  were  rated  by 
military  participants  as  having  relatively  low  priority  in  Situation  1  A.  The  lower  half  of  Figure 
10  shows  that  three  of  these  eight  items  (method  of  fire,  radar/source  unit,  and  asset  damage 
score)  were  rated  unimportant  by  at  least  60%  of  the  military  user  group. 

For  design  engineers.  Figure  3  indicates  that  six  information  items  (asset 
threatened,  TBM  type/subclass,  engagement  status,  TLL,  radar  source/unit,  and  classification/ID) 
were  rated  as  important  to  this  window  for  Situation  1  A.  Figure  8  shows  that  two  of  these  items 
(asset  threatened  and  TBM  type/subclass)  were  deemed  critical  by  at  least  60%  of  the  design 
group. 


The  bottom  half  of  Figure  9  shows  that  eight  items  (lower  tier  coverage,  speed  of 
track,  position  of  track,  heading  of  track,  trajectory  profile/pop-up  graphic,  track  quality,  asset 
damage  score,  and  altitude  of  track)  were  rated  as  least  important  for  the  track  amplification 
window  for  Situation  1  A.  One  item  (lower  tier  coverage)  was  found  to  be  unimportant  by  60% 
of  the  design  group  (see  Figure  1 0). 
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Asset  Threatened 


Track  Amplification  Information  Item  Ratings:  Situation  1A 
General  Monitoring  and  Assessment 


9  Military  Users 
[TTl  Design  Engineers 

A.  Both 

Figure  9.  Data  matrix  for  the  track  amplification  (Situation  1  A). 
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Percent  of  Sample  Rating  Item 
Critical  (High  vs  Low) 


Track  Amplification  Information  Item  Ratings:  Situation  1A 
General  Monitoring  and  Assessment 


%  Military  Users 
mB  Design  Engineers 


Data  matrix  for  the  track  amplification  group  percentages  (Situation  1  A). 
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Classification/ID 


Situation  IB  (situation  assessment  and  monitoring  for  engaged  tracks) 

Looking  at  Figure  11,11  items  were  rated  by  soldiers  as  important  in  the  track 
amplification  window  for  this  situation.  These  items  included  asset  threatened,  threat  priority, 
engagement  status,  type/subclass,  position  of  track,  heading  of  track,  altitude  of  track, 
classification/ID,  TEL,  TLL,  and  status  of  interceptor.  Referring  to  Figure  12,  one  of  these  1 1 
items  (engagement  status)  was  deemed  critical  by  at  least  60%  of  the  military  participants. 

In  contrast,  the  lower  half  of  Figure  1 1  shows  that  eight  information  items  in  the 
track  amplification  window  (method  of  fire,  lower  tier  coverage,  radar/source  unit,  track  quality, 
trajectory  profile/pop-up  graphic,  speed  of  interceptor,  altitude  of  interceptor,  and  heading  of 
interceptor)  were  rated  by  military  participants  as  having  relatively  low  priority  in  Situation  IB. 
The  lower  half  of  Figure  1 0  shows  that  three  of  these  eight  items  (lower  tier  coverage, 
radar/source  unit,  and  track  quality)  were  rated  unimportant  by  at  least  60%  of  the  military 
group. 


The  top  half  of  Figure  1 1  indicates  that  design  engineers  rated  1 1  items  as 
important  in  the  Track  Amplification  Window  to  situation  assessment  and  monitoring  during 
ongoing  target  engagements,  but  none  of  these  items  were  deemed  critical  by  at  least  60%  of  the 
group  (see  Figure  12).  These  1 1  items  included  method  of  fire,  engagement  status,  type/subclass, 
position  of  track,  heading  of  track,  TLL,  time  to  go,  trajectory  profile,  interceptor  position, 
THAAD  missile  ID,  and  status  of  interceptor. 

The  lower  half  of  Figure  1 1  shows  that  nine  information  items  in  the  track 
amplification  window  (TEL,  lower  tier  coverage,  radar/source  unit,  speed  of  track,  estimated 
laimch  point,  classification/ID,  interceptor  speed,  altitude  of  interceptor,  and  heading  of 
interceptor)  were  rated  by  design  engineering  participants  as  having  relatively  low  priority  in 
Situation  IB.  The  lower  half  of  Figine  12  shows  that  two  of  these  nine  items  (altitude  of 
interceptor  and  heading  of  interceptor)  were  rated  unimportant  by  at  least  60%  of  the  design 
group, 

Situation  2  (unexpected  new  threats) 

Looking  at  Figure  13, 10  items  were  rated  by  soldiers  as  important  in  the  track 
amplification  window  for  this  situation.  These  items  included  asset  threatened,  threat  priority, 
engagement  status,  type/subclass,  speed  of  track,  position  of  track,  heading  of  track, 
classification/ID,  TEL,  and  TLL.  Figure  14  shows  that  four  of  these  10  items  (asset  threatened, 
threat  priority,  engagement  status,  and  type/subclass)  were  deemed  critical  by  at  least  60%  of  the 
military  participants. 
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Track  Amplification  Item  Ratings:  Situation  1 B 
General  Monitoring  and  Assessment  -  Engaged  Tracks 
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Figure  11.  Data  matrix  for  the  track  amplification  (Situation  IB). 
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In  contrast,  the  lower  half  of  Figure  13  shows  that  10  information  items  in  the 
track  amplification  window  (method  of  fire,  lower  tier  coverage,  radar/source  unit,  trajectory 
profile/pop-up  graphic,  track  quality,  asset  damage  score,  position  of  interceptor,  speed  of 
interceptor,  altitude  of  interceptor,  and  heading  of  interceptor)  were  rated  by  military 
participants  as  having  relatively  low  priority  in  Situation  2.  The  lower  half  of  Figure  14  shows 
that  one  of  these  10  items  (asset  damage  score)  was  rated  unimportant  by  at  least  60%  of  the 
military  group. 


The  top  half  of  Figure  13  indicates  that  design  engineers  rated  10  information 
items  as  important  in  the  track  amplification  window  to  the  situation  when  new  intelligence 
information  has  been  received  regarding  the  appearance  of  unexpected  new  threats.  These  10 
items  included  method  of  fire,  asset  threatened,  threat  priority,  engagement  status,  TEL,  TLL, 
heading  of  track,  trajectory  profile,  classification/ID,  and  status  of  interceptor.  The  top  half  of 
Figure  14  shows  that  two  of  these  10  items  (asset  threatened  and  classification/ID)  were  rated 
important  by  at  least  60%  of  the  design  group. 

The  lower  half  of  Figure  13  shows  that  design  engineers  rated  nine  information 
items  in  the  track  amplification  window  (lower  tier  coverage,  estimated  laimch  point,  track, 
quality,  THAAD  missile  ID,  time  to  go,  interceptor  position,  interceptor  speed,  altitude  of 
interceptor,  and  heading  of  interceptor)  as  least  important  to  Situation  2.  Figure  14  reveals  that 
four  of  these  nine  items  (lower  tier  coverage,  interceptor  position,  interceptor  speed,  and  altitude 
of  interceptor)  were  rated  unimportant  by  at  least  60%  of  the  design  group. 

Summary 

Figure  15  summarizes  the  military  group  agreement  percentages  presented  in 
Figures  10, 12,  and  14.  Figure  16  summarizes  the  agreement  percentages  for  the  design  group. 

Of  the  important  information  items  indicated  in  the  track  amplification  window  by  military 
users,  seven  items  were  indicated  as  important  across  all  three  situations  (asset  threatened,  threat 
priority,  engagement  status,  position  of  track,  TLL,  classification/ID,  and  heading  of  track). 

Three  additional  items  were  irnportant  in  two  situations  (speed  of  track,  type/subclass,  and 
TEL).  One  item  (engagement  status)  was  deemed  critical  by  at  least  60%  of  the  military 
participants  across  all  three  situations. 

Of  the  information  items  in  the  track  amplification  window,  five  items  (method  of 
fire,  lower  tier  coverage,  reidar/source  unit,  track  quality,  and  trajectory  profile)  were  indicated  by 
military  users  as  relatively  unimportant  across  all  three  situations.  In  addition,  four  items  (asset 
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Track  Amplification  Information  Item  Ratings:  Situation  Two 
New  intelligence  Information:  Review  Engaged  Track(s) 
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0  Military  Users 

Figure  14.  Data  matrix  for  the  track  amplification  group  percentages  (Situation  2).  ^  Design  Engineers 


damage  score,  speed  of  interceptor,  altitude  of  interceptor,  and  heading  of  interceptor^)  were 
noted  as  unimportant  during  two  of  the  situations.  Two  items  (radar/source  unit  and  asset 
damage  score)  were  deemed  relatively  unimportant  across  two  situations  by  at  least  60%  of  the 
military  participants. 


Situation  lA 

Situation  IB 

Situation  2 

Rank  Order 

Method  of  Fire 

-0.6 

-0.2 

-0.2 

2 

Asset  Threatened 

+.4 

+.4 

+.6 

2.5 

Threat  Priority 

+.4 

+.4 

+.6 

2.5 

Type/Subcldss 

0 

+.4 

+.6 

5 

Engagement  Status 

+.6 

+.8 

+.6 

1 

Time  to  Earliest 

Launch 

0 

+.2 

+.4 

8.5 

Lower  Tier  Coverage 

-0.8 

'EOHHI 

24 

TLL 

+.2 

+.2 

+.2 

8.5 

Speed  of  Track 

+.2 

0 

+.2 

10 

Position  of  Track 

+.2 

+.2 

+.4 

7 

Estimated  Launch 

Point 

-0.2 

0 

0 

15 

Heading  of  Track 

+.4 

+.4 

+.2 

5 

Radar/Source  Unit 

-0.6 

-0.6 

-0.2 

24 

Trajectorv  Profile 

-0.2 

-0.2 

18 

Track  Oualitv 

-0.4 

-0.8 

-0.2 

24 

Asset  Damage  Score 

0 

21.5 

Altitude  of  Track 

-0.2 

+.2 

0 

13 

Classification/ID 

+.6 

+.2 

+.2 

5 

Interceptor  Position 

X 

0 

-0.4 

16.5 

Interceotor  Status 

X 

+.2 

0 

11 

Interceptor  Speed 

X 

^-0.4 

-0.4 

21.5 

THAAD  Missile  ID 

X 

0 

0 

13 

Time  to  Go 

X 

0 

0 

13, 

Interceotor  Altitude 

X 

-0.2 

-0.4 

19 

Interceotor  Heading 

X 

-0.2 

-0.2 

16.5 

Figure  15.  Track  amplification  data  window  -  information  item  importance  rating  group 
agreement  percentages  for  military  users. 

For  design  engineers,  two  items  (engagement  status  and  TLL)  were  rated  as 
important  across  all  three  situations.  Seven  additional  items,  (method  of  fire,  asset  threatened, 
type/subclass,  heading  of  track,  trajectory  profile,  classificati’on/ID,  and  interceptor  status)  were 
identified  as  important  across  two  situations.  No  item  was  deemed  critical  by  at  least  60%  of  the 


2^2^.  The  last  three  items  mentioned  here  (speed  of  interceptor,  altitude  of  interceptor,  and  heading  of  interceptor) 
were  only  available  in  two  situations  (Situations  IB  and  2). 
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group  for  all  three  situations.  Design  engineers  rated  one  item  (lower  tier  coverage)  as  relatively 
unimportant  across  all  three  situations.  Six  other  Items  (speed  of  track,  estimated  launch  point, 
track  quality,  interceptor  speed,  interceptor  altitude,  and  interceptor  heading)  were  rated  as 
unimportant  across  two  of  the  situations.  No  item  was  deemed  least  important  by  at  least  60% 
of  the  group  for  all  three  situations. 


Situation  lA 

Situation  IB 

Situation  2 

Rank  Order 

Method  of  Fire 

0 

+.33 

+.33 

6.5 

Asset  Threatened 

+1.0 

0 

+1.0 

1 

Threat  Priority 

0 

0 

+.33 

9 

Tvpe/Subclass 

+.66 

+.33 

0 

3.5 

Engagement  Status 

+.33 

+.33 

+.33 

3.5 

Time  to  Earliest 

Launch 

0 

-0.33 

+.33 

13 

Lower  Tier  Coverage 

-0.66 

-0.33 

-0.66 

24 

ILL 

+.33 

+.33 

+.33 

3.5 

Speed  of  Track 

-0.33 

-0.33 

0 

20 

Position  of  Track 

-0.33 

+.33 

0 

13 

Estimated  Launch 

Point 

0 

-0.33 

-0.33 

20 

Heading  of  Track 

-0.33 

+.33 

+.33 

9 

Radar/Source  Unit 

+.33  , 

-0.33 

0 

13 

Trajectory  Profile 

-0.33 

+.33 

+.33 

9 

Track  Quality 

-0.33 

0 

-0.33 

20 

Asset  Damage  Score 

-0.33 

0 

0 

16.5 

Altitude  of  Track 

.-0.33 

0 

0 

16.5 

Classification/ID 

+.33 

ElOHH 

+.66 

6.5 

Interceptor  Position. 

X 

+.33 

-0.66 

18 

Interceptor  Status 

X 

+.33 

+.33 

3.5 

Interceptor  Speed 

X 

-0.33 

-0.66 

22.5 

THAAD  Missile  ID 

X 

+.33 

-0.33 

13 

Time  to  Go 

X 

+.33 

-0.33 

13 

Interceptor  Altitude 

X 

2.5 

X 

-0.66 

-0.33  1 

22.5 

Figure  16.  Track  amplification  data  window  -  information  item  importance  rating  group 
agreement  percentages  for  design  engineers. 


DISCUSSION  AND  CONCLUSIONS 

The  results  fi’om  this  experiment  clearly  provide  designers  with  some  guidelines  for  the 
design  of  the  THAAD  OSI,  in  particular,  the  engagement  queue  and  track  amplification  windows. 
These  windows  had  previously  been  identified  (in  OSI  Prototype  Version  2.1)  as  areas  to  be 
considered  for  redesign.  This  experiment  provides  specific  insight  into  the  redesign  of  these 
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windows.  The  rank  orders  displayed  in  Figure  7  for  the  engagement  queue  and  Figure  15  for  the 
track  amplification  data  window  provide  the  basis  for  this  design  guidance. 

The  engagement  queue  window,  currently  designed  as  a  table  of  rows  and  columns, 
provides  the  THAAD  engagement  operator  with  various  information  items  about  incoming 
threats,  both  engaged  and  unengaged.  Each  track  is  displayed  as  a  separate  row,  and  ten  column 
headings  identify  various  information  about  each  track.  Currently,  operators  may  need  to  scan 
extensively  or  scroll  through  the  window  in  order  to  find  information  not  displayed  on  the  first 
page  of  the  window.  Two  information  items  in  this  window  (asset  threatened  and  TBM 
type/subtype)  have  been  identified  in  this  study  as  being  the  most  importation  items  across 
multiple  representative  situations.  As  such,  ,  designers  now  know  to  place  these  highly  used 
items  in  a  prominent  area  of  the  display  window  (e.g.,  left-most  columns  of  the  display),  limiting 
the  amount  of  unnecessary  scanning  or  searching  that  operators  need  to  perform  to  locate  them. 
In  contrast,  lower  tier  coverage  and  method  of  fire  were  identified  as  having  the  lowest  priorities. 
These  items  can  then  be  delegated  to  less  prominent  areas  of  the  engagement  queue  window. 

The  track  amplification  window  is  also  currently  displayed  as  a  table  showing  detailed 
mformation  for  a  “hooked”  track.  Information  items  are  displayed  on  different  rows  (18  for 
unengaged  tracks  and  25  for  engaged  tracks).  Again,  operators  need  to  scroll  through  the  window, 
searching  for  information  that  is  not  displayed  on  the  first  page.  Seven  of  these  information 
items  (asset  threatened,  threat  priority,  engagement  status,  TBM  type/subclass,  classification/ID, 
position  of  tracks  and  heading  of  track)  were  indicated  in  this  study  as  being  important  across  all 
situations  tested.  Because  these  seven  items  are  used  often,  especially  threat  priority  and 
engagement  status,  designers  need  to  place  them  in  a  prominent  area  of  the  display  window, 
specifically  on  the  first  page,  limiting  the  amount  of  additional  scrolling,  scanning,  or  mouse  clicks 
that  operators  need  to  perform  to  locate  them.  In  addition,  speed  of  track,  TEL,  and  TLL,  should 
also  be  given  consideration  in  their  placement  because  they  were  found  to  be  important  in  at  least 
two  of  the  three  situations  tested. 

.In  contrast,  nine  mformation  items  (method  of  fire,  lower  tier  coverage,  radar/source  unit, 
track  quality,  asset  damage  score,  speed  of  interceptor,  altitude  of  interceptor,  heading  of 
interceptor,  and  trajectory  profile/pop-up  graphic)  were  identified  as  having  the  lowest  priorities. 
These  items  can  then  be  delegated  to  less  prominent  areas  of  the  track  amplification  window.  In 
addition,  the  implementation  of  the  trajectory  profile/pop-up  graphic  needs  to  be  reconsidered. 

As  currently  implemented,  this  graphic  is  accessed  by  “clicking”  on  a  button  in  the  track 
amplification  window.  This  graphic  is  then  displayed  in  the  window,  displacing  all  the  other 
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information  displayed  by  rows.  Since  this  graphic  was  designated  as  relatively  unimportant,  it 
should  not  displace  information  that  is  more  important. 

In  addition,  although  these  windows  are  currently  implemented  as  tables  which  operators 
need  to  scroll  through,  these  implementations  are  also  being  evaluated  and  other  possible 
implementations  being  considered.  The  results  of  this  study  still  give  designers  guidelines  that 
are  independent  of  the  final  implementation  chosen.  Information  items  found  to  be  important  in 
the  engagement  queue  and  track  amplification  windows  need  to  be  placed  in  a  prominent  area  of 
the  display,  whether  displayed  in  a  table,  graphically,  etc.  This  information  needs  to  be 
presented  to  the  operators  requiring  very  little  extra  action  on  their  part  (i.e.,  scrolling,  mouse 
clicks,  scanning,  etc.).  Information  deemed  as  relatively  unimportant  (e.g.,  information  that  is 
“nice”  to  have  available  but  not  needed  in  most  cases)  can  be  displayed  less  prominently. 

One  interesting  result  of  this  research  relates  to  the  amoimt  of  agreement  in  item  ratings 
between  military  users  and  system  design  engineers.  Comparing  the  group  rank  orderings  listed 
in  Figures  7  and  8  for  the  engagement  queue,  when  one  considers  values  attributable  to  tied  ranks, 
one  finds  a  positive  amount  of  similarity  in  rankings  (Ganuna  rank-order  correlation  statistic  = 
.5238;  p  =  .04^)  for  all  the  information  items  except  two.  In  fact,  there  was  perfect  agreement  for 
the  most  important  and  the  two  least  important  information  items  within  the  engagement  queue. 
The  two  items  that  received  the  most  divergent  ratings  between  the  two  groups  were  Engagement 
Status  and  TLL.  The  design  engineers  consistently  rated  these  two  items  as  relatively  more 
important  (ranks  2  and  3)  whereas  users  see  these  two  items  as  relatively  unimportant  (ranks  7 
and  8). 

When  comparing  group  rankings  for  the  information  in  the  Track  Amplification  Data 
Window,  the  data  in  Figures  15  and  16  show  a  fair  amount  of  correspondence  (Gamma  rank-order 
correlation  statistic  =  .6273;  p  =  .000039)  between  groups  for  10  of  the  items.  These  items  are 
asset  threatened,  type/subclass,  engagement  status,  lower  tier  coverage,  track  quality, 
classification/ID,  interceptor  position,  interceptor  speed,  THAAD  missile  ID,  and  time  to  go. 
There  was  considerable  divergence  in  group  ratings  for  three  items.  These  items  were  method  of 
fire,  radar/source  unit,  and  speed  of  track.  Military  users  consistently  rate  the  first  two  items  as 
relatively  less  important,  but  designers  rate  them  as  important.  The  last  item,  speed  of  track,  is 
rated  more  important  by  users  than  by  design  engineers. 


^Gamma.  The  Ganuna  non-paiametric  correlation  coefficient  (variables  in  ranks)  is  preferable  to  Spearman  R  or 
Kendall  tau  when  the  data  contain  many  tied  observations  (Siegel  &  Castellan,  1988). 


29 


The  fact  that  system  designers  differ  in  their  ratings,  at  least  to  some  degree,  from  the 
ratings  of  system  users  indicates  that  users  must  be  involved  in  the  system  design  process. 
Second,  this  involvement  must  be  early  in  the  design  process.  User  design  input,  as  documented 
in  this  report,  can  be  objectively  obtained.  Finally,  these  objective  data  must  then  be  provided  to 
the  system  design  engineers  to  influence  features  of  the  final  human  computer  interface. 
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